Guard eager OpenFeature init and add detailed /ffe/evaluate for Go agentless configuration contract by pavlokhrebto · Pull Request #7580 · DataDog/system-tests

pavlokhrebto · 2026-08-25T13:07:11Z

Motivation

FFL-2695's Go agentless configuration-source implementation has now landed in dd-trace-go (DataDog/dd-trace-go#5223, #5226, #5346, #5363), so this PR brings the Go parametric app up to the /ffe/start and /ffe/evaluate contract already implemented for Python (#7411) and declares the coverage as supported.

Changes

main.go — lazy activation: skip eager OpenFeature provider init when any DD_FEATURE_FLAGS_* configuration variable is set, leaving setup to /ffe/start, since the contract forbids configuration delivery before the provider is accessed. Tests predating that contract keep the eager path, which stays asynchronous on purpose: it runs for every parametric test that sets no Feature Flagging variable, and blocking there would add the 10s DD_EXPERIMENTAL_FLAGGING_PROVIDER_INITIALIZATION_TIMEOUT_MS to each container start.

ffe.go

/ffe/start now waits for Init before answering. Plain SetProvider returns first, so the endpoint used to answer 200 with no configuration and the next evaluation got the default.
The wait carries our own 15s deadline via SetProviderWithContextAndWait rather than SetProviderAndWait's background context, so tracers that only bound Init cannot hang the suite — this is what keeps the prod job green against released tracers.
PROVIDER_NOT_READY is treated as a successful start: the provider is registered and evaluations return defaults until configuration arrives.
Start state is guarded by a mutex plus a success flag instead of sync.Once, so a failed attempt is not cached as success.
/ffe/evaluate uses the *ValueDetails APIs per variationType and returns {value, reason, errorCode}. Details are preserved on the error path too — the SDK returns them alongside the error, and _ffe_evaluate_with_rc_retry keys on errorCode == PROVIDER_NOT_READY to decide whether to retry.

tests/parametric/test_ffe/test_configuration_sources.py — _remote_config_products no longer filters on post_only. dd-trace-go polls /v0.7/config with GET where every other library uses POST, so all of its RC requests were discarded and FFE_FLAGS never appeared in the products set. The verb says nothing about which products the body advertises, and the capability assertion beside it was already unfiltered.

manifests/golang.yml — declare v2.11.0-dev for tests/parametric/test_ffe/test_configuration_sources.py (replacing the file-level block and ten class/test entries) and for tests/ffe/test_agentless_configuration.py. All 22 configuration-source tests pass on Linux; the end-to-end test was verified enforced and green.

Workflow

⚠️ Create your PR as draft ⚠️
Work on you PR until the CI passes
Mark it as ready for review
- Test logic is modified? -> Get a review from RFC owner.
- Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
A docker base image is modified?
- the relevant build-XXX-image label is present
A scenario is added, removed or renamed?
- Get a review from R&P team

…entless configuration contract

github-actions · 2026-08-25T13:07:38Z

CODEOWNERS have been resolved as:

manifests/golang.yml                                                    @DataDog/system-tests-reviewers
tests/parametric/test_ffe/test_configuration_sources.py                 @DataDog/feature-flagging-and-experimentation-sdk
utils/build/docker/golang/parametric/ffe.go                             @DataDog/system-tests-reviewers
utils/build/docker/golang/parametric/main.go                            @DataDog/system-tests-reviewers

…e and endpoint (#5218) ### What does this PR do? Lays the groundwork for Agentless Feature Flags delivery (polling configuration directly from Datadog over HTTPS instead of via the Agent's Remote Config): five new `DD_FEATURE_FLAGS_*` environment variables, `internal/config` fields/accessors for them, the source-precedence resolution logic (`internal/openfeature.ResolveSettings`/`RemoteConfigSourceSelected`), and Agentless endpoint URL resolution (`openfeature.buildAgentlessEndpoint`, managed vs. custom). No behavior change yet — nothing is wired up to the provider or issues any network requests. This is Phase 1 of a multi-PR rollout; the poll loop, JSON:API envelope parsing, and provider wiring/activation follow in subsequent PRs. ### Stack 1. #5218 — settings, source precedence, endpoint (this PR) 2. #5222 — JSON:API parser, poll loop 3. #5223 — provider wiring, activation, lifecycle, RC gating 4. #5226 — init outcomes, provider events ### Motivation dd-trace-go is one of the last server SDKs to support Agentless Feature Flags delivery (Python, Node, and Java have shipped it; .NET is in review; Ruby is also pending). Agentless removes the Agent as a dependency and becomes the default delivery mechanism. FFL-2695. ### Reviewer's Checklist - [x] Changed code has unit tests for its functionality at or near 100% coverage. - [x] [System-Tests](https://github.com/DataDog/system-tests/) covering this feature have been added and enabled with the va.b.c-dev version tag: DataDog/system-tests#7580 - [ ] There is a benchmark for any new code, or changes to existing code. - [ ] If this interacts with the agent in a new way, a system test has been added. - [x] New code is free of linting errors. You can check this by running `make lint` locally. - [x] New code doesn't break existing tests. You can check this by running `make test` locally. - [ ] Add an appropriate team label so this PR gets put in the right place for the release notes. - [x] All generated files are up to date. You can check this by running `make generate` locally. - [ ] Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild. Make sure all nested modules are up to date by running `make fix-modules` locally. Unsure? Have a question? Request a review! Co-authored-by: pavlo.khrebto <pavlo.khrebto@datadoghq.com>

### What does this PR do? Adds the Agentless configuration delivery mechanism itself, stacked on #5218 (source resolution/endpoint): a JSON:API envelope parser for the Universal Flags Configuration payload (`openfeature/jsonapi.go`), and the poll loop that fetches it (`openfeature/agentless_source.go`) — per-poll retries with backoff, ETag/304 handling with last-known-good semantics, gzip decoding, and per-category log-once for persistent failures. Includes an `httptest.Server`-based fake backend mirroring the system-tests mock (`utils/mocked_backend/ffe.py`) covering the full behavior matrix: ETag not advanced on parse failure, retry-within-poll, non-retryable failures, no-overlap between polls, and no log line ever containing the endpoint URL. Still not wired up to the provider — nothing issues a request until Phase 3 lands. ### Stack 1. #5218 — settings, source precedence, endpoint 2. #5222 — JSON:API parser, poll loop (this PR) 3. #5223 — provider wiring, activation, lifecycle, RC gating 4. #5226 — init outcomes, provider events ### Motivation Continuation of Agentless Feature Flags delivery (FFL-2695); see #5218 for the overall context. ### Reviewer's Checklist - [x] Changed code has unit tests for its functionality at or near 100% coverage. - [x] [System-Tests](https://github.com/DataDog/system-tests/) covering this feature have been added and enabled with the va.b.c-dev version tag: see DataDog/system-tests#7580 (referenced from #5218). - [ ] There is a benchmark for any new code, or changes to existing code. - [ ] If this interacts with the agent in a new way, a system test has been added. - [x] New code is free of linting errors. You can check this by running `make lint` locally. - [x] New code doesn't break existing tests. You can check this by running `make test` locally. - [ ] Add an appropriate team label so this PR gets put in the right place for the release notes. - [x] All generated files are up to date. You can check this by running `make generate` locally. - [ ] Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild. Make sure all nested modules are up to date by running `make fix-modules` locally. Unsure? Have a question? Request a review! Co-authored-by: pavlo.khrebto <pavlo.khrebto@datadoghq.com>

…ss delivery (#5223) ### What does this PR do? Wires up the provider itself, stacked on #5222 (poll loop/parser): `NewDatadogProvider` now resolves the delivery source and branches to Remote Config, Agentless, or a `NoopProvider`, replacing the old `DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED`-gated check. Adds the Agentless activation critical section (`startWithAgentless`) with a `shutdownCalled` re-check so a poller can never be registered after `Shutdown` — that would otherwise leak a billable poller for the process lifetime. Reorders `ShutdownWithContext` so teardown runs without holding the provider's lock (needed because `agentless.Stop` joins the poll goroutine, which itself calls `updateConfiguration` and takes the same lock). Gates the tracer's eager Remote Config subscribe on the resolved source (`ddtrace/tracer/remote_config.go`). Also fixes a real pre-existing bug found by a new concurrency test: concurrent `Init` calls could double-start a writer and panic; `Init` is now idempotent. Updates `openfeature/doc.go` with the new environment variables and the delivery-source model. ### Stack 1. #5218 — settings, source precedence, endpoint 2. #5222 — JSON:API parser, poll loop 3. #5223 — provider wiring, activation, lifecycle, RC gating (this PR) 4. #5226 — init outcomes, provider events ### Motivation Continuation of Agentless Feature Flags delivery (FFL-2695); see #5218 for the overall context. ### Reviewer's Checklist - [x] Changed code has unit tests for its functionality at or near 100% coverage. - [x] [System-Tests](https://github.com/DataDog/system-tests/) covering this feature have been added and enabled with the va.b.c-dev version tag: see DataDog/system-tests#7580 (referenced from #5218). - [ ] There is a benchmark for any new code, or changes to existing code. - [ ] If this interacts with the agent in a new way, a system test has been added. - [x] New code is free of linting errors. You can check this by running `make lint` locally. - [x] New code doesn't break existing tests. You can check this by running `make test` locally. - [ ] Add an appropriate team label so this PR gets put in the right place for the release notes. - [x] All generated files are up to date. You can check this by running `make generate` locally. - [ ] Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild. Make sure all nested modules are up to date by running `make fix-modules` locally. Unsure? Have a question? Request a review! Co-authored-by: kakkoyun <kakkoyun@users.noreply.github.com> Co-authored-by: leo.romanovsky <leo.romanovsky@datadoghq.com>

…5226) ### What does this PR do? Final PR in the Agentless Feature Flags delivery series (stacked on #5223). Adds `DD_EXPERIMENTAL_FLAGGING_PROVIDER_INITIALIZATION_TIMEOUT_MS` (default 10s) replacing the hardcoded 30s init timeout, and implements the three `InitWithContext` outcomes: a permanent delivery failure returns a `*openfeature.ProviderInitError` immediately, configuration already present returns `nil`, and a timeout with delivery still running returns `nil` + a warning instead of the raw `ctx.Err()` (a deliberate change from the old Remote-Config-only behavior — see below). Adds `openfeature/events.go`: `EventChannel()` returns the same stored channel on every call (required by the SDK's listener), emitting `ProviderReady` once on the first configuration, `ProviderConfigChange` on later ones, and `ProviderStale` when configuration goes nil (e.g. an empty Remote Config set) so evaluation status reflects reality. `ProviderReady` is guaranteed to survive a full event buffer via a non-blocking drain-and-retry. ### Stack 1. #5218 — settings, source precedence, endpoint 2. #5222 — JSON:API parser, poll loop 3. #5223 — provider wiring, activation, lifecycle, RC gating 4. #5226 — init outcomes, provider events (this PR) ### Motivation Completes Agentless Feature Flags delivery (FFL-2695); see #5218 for the overall context. **Behavior change to flag for cross-language sign-off:** on an `Init` timeout with delivery still running, the provider now returns `nil` instead of `ctx.Err()`. Go's `ErrorState` doesn't block flag evaluation, so the only customer-visible difference is the reported status/event — but this could affect a health check that inspected the old error return. ### Reviewer's Checklist - [x] Changed code has unit tests for its functionality at or near 100% coverage. - [x] [System-Tests](https://github.com/DataDog/system-tests/) covering this feature have been added and enabled with the va.b.c-dev version tag: see DataDog/system-tests#7580 (referenced from #5218). - [ ] There is a benchmark for any new code, or changes to existing code. - [ ] If this interacts with the agent in a new way, a system test has been added. - [x] New code is free of linting errors. You can check this by running `make lint` locally. - [x] New code doesn't break existing tests. You can check this by running `make test` locally. - [ ] Add an appropriate team label so this PR gets put in the right place for the release notes. - [x] All generated files are up to date. You can check this by running `make generate` locally. - [ ] Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild. Make sure all nested modules are up to date by running `make fix-modules` locally. Unsure? Have a question? Request a review! Co-authored-by: leoromanovsky <leo.romanovsky@datadoghq.com>

…ation

…ebto/FFL-2695/go-agentless-configuration

…polls by verb

…ithub.com:DataDog/system-tests into pavlo.khrebto/FFL-2695/go-agentless-configuration

datadog-official · 2026-09-11T21:05:23Z

✅ All CI checks and tests passed.

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: a21571b | Docs | View more details | Give us feedback!}

…ebto/FFL-2695/go-agentless-configuration

…/start

…ation

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6b27857e0b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

…fe/start to retry

…ithub.com:DataDog/system-tests into pavlo.khrebto/FFL-2695/go-agentless-configuration

… v2.11.0-dev

Copilot

🟡 Changes recommended

Three unresolved moderate issues affect provider initialization and enabled coverage.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Updates Go Feature Flagging support with lazy provider initialization, detailed evaluations, GET remote-config detection, and enabled coverage.

Changes:

Adds guarded, bounded provider startup.
Returns evaluation details and error codes.
Enables Go configuration-source tests.

File summaries

File	Summary	Final review status
`utils/build/docker/golang/parametric/main.go`	Adds lazy-init environment checks.	Moderate issue (2 votes): an existing remote-config test may not initialize the provider.
`utils/build/docker/golang/parametric/ffe.go`	Implements provider startup and detailed evaluations.	Moderate issue (1 vote): the default path may reinitialize the provider.
`tests/parametric/test_ffe/test_configuration_sources.py`	Detects FFE products across request methods.	No final comments.
`manifests/golang.yml`	Enables Go FFE configuration-source coverage.	Moderate issue (1 vote): the activated shared handler may evaluate before initialization completes.

Review details

Suppressed comments (2)

manifests/golang.yml:1330

This activation exercises utils/build/docker/golang/app/_shared/common/ffe.go, not the parametric ffeStart changed above. The shared handler still calls plain of.SetProvider, which returns before agentless Init, and immediately evaluates the first /ffe request; that request can therefore return the default before UFC is loaded. Update the shared handler to wait for initialization (or defer this declaration until it does).

  tests/ffe/test_agentless_configuration.py: v2.11.0-dev

utils/build/docker/golang/parametric/ffe.go:34

When no DD_FEATURE_FLAGS_* variables are set, newServer has already populated s.ofClient through the eager path, but ffeStarted is still false. The default-environment Test_Parametric_FFE_Start now constructs and installs a second provider here, unnecessarily reinitializing the global provider and potentially waiting for the initialization timeout, so the eager compatibility path is no longer preserved.

		if ffeStarted {
			return nil
		}

Files reviewed: 4/4 changed files
Comments generated: 1
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

pavlokhrebto · 2026-09-17T13:43:56Z

/code blockers

gh-worker-devflow-routing-ef8351 · 2026-09-17T13:43:59Z

View all feedbacks in Devflow UI.

2026-09-17 13:43:59 UTC ℹ️ Start processing command /code blockers

2026-09-17 13:44:00 UTC ℹ️ Devflow:

Checking merge blockers for #7580...

2026-09-17 13:44:08 UTC ℹ️ Devflow: /code blockers

Detected 1 merge blocker(s) to address:

🟠 Pending

Merge gate rule reviewers-approval is running: 1 approval missing
All required reviewers must approve this pull request before it can be merged. Learn more in our FAQ.
- Needs approval from Team: feature-flagging-and-experimentation-sdk
Hint: Questions about this check? Reach out in #dx-source-code-management on Slack.

Guard eager OpenFeature init and add detailed /ffe/evaluate for Go ag…

0c4b674

…entless configuration contract

pavlokhrebto changed the title ~~Guard eager OpenFeature init and add detailed /ffe/evaluate for Go ag…~~ Guard eager OpenFeature init and add detailed /ffe/evaluate for Go agentless configuration contract Aug 25, 2026

pavlokhrebto added 4 commits September 11, 2026 08:37

Merge branch 'main' into pavlo.khrebto/FFL-2695/go-agentless-configur…

ddc8e97

…ation

Merge branch 'main' of github.com:DataDog/system-tests into pavlo.khr…

7bfaebc

…ebto/FFL-2695/go-agentless-configuration

fix(golang): block /ffe/start on provider init and stop filtering RC …

3d5729f

…polls by verb

Merge branch 'pavlo.khrebto/FFL-2695/go-agentless-configuration' of g…

f258148

…ithub.com:DataDog/system-tests into pavlo.khrebto/FFL-2695/go-agentless-configuration

chore: re-run CI against the tracer fix branch

136dd5e

pavlokhrebto added 3 commits September 14, 2026 09:27

Merge branch 'main' of github.com:DataDog/system-tests into pavlo.khr…

f5cc36b

…ebto/FFL-2695/go-agentless-configuration

fix(golang/parametric): treat PROVIDER_NOT_READY as a successful /ffe…

b9bbe44

…/start

fix(golang/parametric): bound the /ffe/start wait with our own deadline

b56ca81

Merge branch 'main' into pavlo.khrebto/FFL-2695/go-agentless-configur…

6b27857

…ation

pavlokhrebto marked this pull request as ready for review September 15, 2026 06:38

pavlokhrebto requested review from a team as code owners September 15, 2026 06:38

pavlokhrebto requested review from danyal002 and sameerank and removed request for a team September 15, 2026 06:38

pavlokhrebto requested review from Anilm3 and leoromanovsky and removed request for danyal002 September 15, 2026 06:38

chatgpt-codex-connector Bot reviewed Sep 15, 2026

View reviewed changes

Comment thread utils/build/docker/golang/parametric/ffe.go Outdated

Comment thread utils/build/docker/golang/parametric/ffe.go Outdated

pavlokhrebto added 3 commits September 15, 2026 10:21

fix(golang/parametric): keep evaluation details on error and allow /f…

d0f849d

…fe/start to retry

Merge branch 'pavlo.khrebto/FFL-2695/go-agentless-configuration' of g…

30eaae6

…ithub.com:DataDog/system-tests into pavlo.khrebto/FFL-2695/go-agentless-configuration

feat(golang): declare FFE agentless configuration-source support from…

a21571b

… v2.11.0-dev

pavlokhrebto requested a lite review from Copilot September 16, 2026 06:27

Copilot started reviewing on behalf of pavlokhrebto September 16, 2026 06:28 View session

Copilot AI reviewed Sep 16, 2026

View reviewed changes

Comment thread utils/build/docker/golang/parametric/main.go

pavlokhrebto requested a review from a team September 16, 2026 09:14

nccatoni approved these changes Sep 17, 2026

View reviewed changes

leoromanovsky approved these changes Sep 18, 2026

View reviewed changes

Comment thread manifests/golang.yml

gh-worker-dd-devflow-36fce6 Bot added mergequeue-status: waiting mergequeue-status: removed mergequeue-status: queued mergequeue-status: in_progress and removed mergequeue-status: waiting mergequeue-status: removed mergequeue-status: queued labels Sep 18, 2026

gh-worker-dd-mergequeue-cf854d Bot merged commit 27cc8ba into main Sep 18, 2026
444 checks passed

gh-worker-dd-devflow-36fce6 Bot removed the mergequeue-status: in_progress label Sep 18, 2026

gh-worker-dd-mergequeue-cf854d Bot deleted the pavlo.khrebto/FFL-2695/go-agentless-configuration branch September 18, 2026 08:56

gh-worker-dd-devflow-36fce6 Bot added the mergequeue-status: done label Sep 18, 2026

Navigation Menu

Guard eager OpenFeature init and add detailed /ffe/evaluate for Go agentless configuration contract - #7580

Guard eager OpenFeature init and add detailed /ffe/evaluate for Go agentless configuration contract#7580
gh-worker-dd-mergequeue-cf854d[bot] merged 13 commits into
mainfrom
pavlo.khrebto/FFL-2695/go-agentless-configuration

pavlokhrebto commented Aug 25, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 25, 2026 •

edited

Loading

Uh oh!

datadog-official Bot commented Sep 11, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

pavlokhrebto commented Sep 17, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Sep 17, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

pavlokhrebto commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Workflow

Reviewer checklist

Uh oh!

github-actions Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-official Bot commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (2)

Uh oh!

Uh oh!

pavlokhrebto commented Sep 17, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟠 Pending

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pavlokhrebto commented Aug 25, 2026 •

edited

Loading

github-actions Bot commented Aug 25, 2026 •

edited

Loading

datadog-official Bot commented Sep 11, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Sep 17, 2026 •

edited

Loading